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We prove a functional limit theorem for the rescaled occupa- 
tion time fluctuations of a (d, a, /3)-branching particle system [parti- 
cles moving in R'' according to a symmetric a-stable Levy process, 
branching law in the domain of attraction of a (1 + /3)-stable law, 
< /3 < 1, uniform Poisson initial state] in the case of intermediate 
dimensions, a/ P < d < a{l + 13) / (3. The limit is a process of the form 
K\^, where if is a constant, A is the Lebesgue measure on R'', and 
^ = (^t)t>o is a (1 -I- /3)-stable process which has long range depen- 
dence. For Q < 2, there are two long range dependence regimes, one 
for P > d/{d + a), which coincides with the case of finite variance 
branching (/3 = 1) , and another one for P < d/{d + a), where the long 
range dependence depends on the value of /3. The long range depen- 
dence is characterized by a dependence exponent k which describes 
the asymptotic behavior of the codifference of increments of ^ on in- 
tervals far apart, and which is d/a for the first case (and for a = 2) 
and {1 + P — d/{d + a))d/a for the second one. The convergence proofs 
use techniques of iS'(R'*)-valued processes. 



1. Introduction. We consider a so-called {d, a, /3)-branching particle sys- 
tem in M/^ described as follows. Particles start off at time t = from a Poisson 
random field with intensity measure A = (Lebesgue measure), and they 
evolve independently, moving according to a standard symmetric Q-stable 
Levy process (0 < a < 2) and splitting at rate V according to the branching 
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law 



Pk = 




k = 0,2,3,... 



k = l 



< (3 < 1. This law is critical, for /? = 1, it is binary branching, and for 
/3 < 1, it is a simple typical element of the domain of attraction of a stable 
law with exponent 1 + /?. Its generating function is 



This branching particle system and its associated superprocess have been 
widely studied; some of the early results appear in [9, 10, 11, 14, 17, 19, 20] 
and references therein. In this paper we prove a functional limit theorem for 
the rescaled occupation time fluctuations of the system with /3 < 1 in the 
case of "intermediate dimensions," a/ (3 < d < a{l + P)/f3, where the limit 
process has long range dependence (Theorem 2.2), and we show that for 
a <2 there are two different long range dependence regimes, depending on 
whether f3 is above or below the value d/{d + a) (Theorem 2.7). 

Let us recall first the result for (3 = 1, where the branching law has finite 
variance. A functional limit theorem for the occupation time fiuctuations of 
the system was proven in [6] when the limit process has long range depen- 
dence, which occurs for the intermediate dimensions 



The limit process is of the form CXC, where C is a constant and C = iCt)t>o 
is a real valued, continuous, self-similar, long range dependence Gaussian 
process, called sub-fractional Brownian motion, whose covariance function 
is 



where h = 3 — d/a. The long range dependence of ^ is characterized by 
the behavior of the covariance of increments on intervals separated distance 
T, which decays like T""'/" as T — > oo. Properties of C are studied in [5]. 
The reason for the name sub-fractional Brownian motion is the fact that the 
increments of ^ on nonoverlapping intervals are more weakly correlated than 
those of fractional Brownian motion, whose covariance function is ^[s^ + 
t^ — \s — tl'^], and their covariance decays faster as the distance between 
the intervals tends to oo; in this sense sub- fractional Brownian motion is 
intermediate between Brownian motion and fractional Brownian motion. 

The Gaussian property of the limit process in [6] is due to the finiteness 
of the variance of the binary branching law. More general critical, finite 
variance branching laws lead to essentially the same limit process [23]. A 





0<s<l. 




a < d < 2a. 



/ + t'^-i[(s + t)^ + |s-t|''] 
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substantially different and more interesting situation occurs with infinite 
variance branching, /3 < 1. There are new technical problems in the proofs 
and the results reveal new properties of the (d, a, /3)-branching particle sys- 
tem. Condition (1.2) for long range dependence is now replaced by 

a.3) i<-^ 

(note that a/(3 <d is the condition for the system to converge for large 
time toward an equilibrium state which has intensity A; for d < a//3, the 
system goes to local extinction [19]). The occupation time fluctuation limit 
process for (3 <1 resembles the one for /3 = 1 in that it has a simple spatial 
structure and a complicated temporal one. It has the form K\^^ where K is a 
constant and ^ = {S,t)t>o is a continuous, self-similar, long range dependence 
(1 + /?)-stable process, which may be called a sub-fractional stable process by 
analogy with the case P = 1 [see Remark 2.4(a)]. The convergence takes place 
in the space of continuous functions C([0,r],5'(M^)) for any r > 0, where 
S'{W^) is the space of tempered distributions, dual of the space 5(]R ) of 
smooth rapidly decreasing functions. We stress that convergence methods 
for 5' (M"')- valued processes play a fundamental role in this paper due to 
the convenient topology of 5'(R"') (although here all infinite dimensional 
processes are measure- valued). 

Concerning functional convergence of the occupation time fluctuation pro- 
cess, in the finite variance case the tightness proof in [6] employed standard 
methods based on moment estimates. For /3 < 1, there are no moments of 
orders > 1 + /3 and a more delicate approach is needed. The general scheme 
for the identification of a unique limit is similar to that in [6], which in- 
volves a space-time random field method introduced in [4] . This approach is 
simpler in the present case than proving convergence of finite dimensional 
distributions, but additional technical work is needed to handle /? < 1, due 
to the fact that the Fourier transform method that was widely used in [6] is 
not applicable in most cases in this paper. 

The long range dependence of the process ^ is characterized by means 
of the asymptotic behavior of the codifference of increments on intervals 
distance T apart as T ^ oo, and this is given in terms of a dependence 
exponent k (Definition 2.5). Regarding codifference, see [25]. For a = 2, the 
dependence exponent is k = d/a, and the codifference decays at the same 
rate as the covariance in the finite variance case (/? = 1), that is, T^"^/" [5]. 
For a <2, there are two long range dependence regimes which are separated 
by the value P = d/{d + a): 

(1) For d/{d + a) < 13 < 1, the dependence exponent is k = d/a. 

(2) For P <d/{d + a), the dependence exponent depends on /? as follows: 
K = (1 + /? - d/{d + a))d/a. 
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Our methods are mainly analytic and they have technical advantages, 
but "physical" explanations of the results in terms of the particle system 
are not apparent from them. The simple spatial structure of the limit pro- 
cess (Lebesgue measure) arises already in occupation time fluctuations of a 
Poisson system of independent Brownian motions in dimension d = 1 [15] 
[Theorem 0.4(i)], and the same thing happens with critical binary branch- 
ing two dimensions higher, d = 3 [9] (Theorem 1), which is the special case 
of (1.3) for a = 2, (3 = 1 (see also [12] for a more general setup which also 
shows the shift in dimensions from nonbranching to branching; [9] and [12] 
have only single time limits). The cause of the long range dependence with 
[3 = 1 was attributed intuitively in [6] to "clan recurrence." A clan is a 
family of particles with a common ancestor. Clan recurrence means that 
any ball is visited by clans infinitely often and at arbitrarily large times, 
each visit adding a random amount to the occupation time of the ball. This 
phenomenon was investigated in [29] for the branching particle system in 
equilibrium with a = 2, (3 = 1 (and a partial result for /? < 1); it was shown 
that clan recurrence holds in dimensions (i = 3,4. The (d, a, /3)-branching 
system with initial Poisson condition is not in equilibrium. However, based 
on [6, 7, 8, 29], and the results in the present paper, the natural conjectures 
are that clan recurrence holds for the general (d, a, /3)-branching system if 
and only if a/ [3 < d < a{l + (3)/(3, with equilibrium or some other initial 
conditions (including Poisson), and that this is the cause of the long range 
dependence of the occupation time fluctuation limit process. But at the 
borderline d = a(l + f3)/f3, the long range dependence disappears [7, 8]. We 
would like to have some intuitive explanation for the two long range depen- 
dence regimes separated by (3 = d/{d + a) (whereas no such phenomenon 
occurs in the Brownian case, a = 2), but this still remains an enigma to 
us. A related curious fact is that the dependence exponent k converges to 
d/{d + a) as /3\0. 

We mention some other results on occupation times. As already stated, 
for d< a/ (3, the system becomes extinct locally. However, for d = a/ (3, there 
is a functional ergodic theorem [30] . With (3 = 1, there are functional limit 
theorems for the fluctuations in dimensions d>2a, where there is no long 
range dependence [7]. In [3] the fluctuations of the occupation time of the 
origin are studied for a critical binary branching random walk on the d- 
dimensional lattice, d>3; the convergence results are parallel to those in 
[6] and [7], but the proofs are quite different. [3] treats also the case of the 
branching random walk in equilibrium (see also [5] for the case /? = 1 in 
equilibrium, where only covariance calculations were done). The results of 
[6] and [7] are extended in [23] for general critical, finite variance branch- 
ing, with both equilibrium and Poisson initial conditions. [6] and [7] contain 
references to other relevant papers, among them those that awakened our 
interest in this subject, [9], [15] and [20] (although they do not refer to long 
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range dependence). In [8] we study the occupation time fluctuations of the 
(d, a, /3)-branching particle system with /3 < 1 and Poisson initial condition 
in dimensions d > a(l + /?)//?, where there is no long range dependence. 
For the "critical" dimension, d = a(l + (3)/ (3, the limit process has the form 
KXr], where r/ = {r]t)t>o is a real (1 + /3)-stable process with stationary in- 
dependent increments. For "large" dimensions, d > a{l + the limit is 
an 5'(M'^)-valued (not measure-valued) (1 + /3)-stable process with station- 
ary independent increments. The size of the occupation time fluctuations 
is given by the norming for convergence. If T is the time scaling parame- 
ter and Ft denotes the norming [see (1.4) below]. Ft is T'''^+/^-Wo')l^)/ii+l^) 
for a/l3<d<a{l + P)/P, (riogr)i/a+/3) iov d = a{l + P)/ /3, and T^a+Z^) 
for d > a(l + (3)/ (3. This illustrates the phenomena that atypical normings 
are associated with long range dependence, and higher order fluctuations 
(involving a logarithmic factor) arise at the borderline between different be- 
haviors. The norming for high dimensions is the one for the classical central 
limit theorem. Several unsolved questions of "physical" interpretation are 
also brought up in [8], including the ones mentioned above. 

Long range dependence is now an area of intensive research due to its 
mathematical appeal and its manyfold applications (see, e.g., [16]). In par- 
ticular, there are other long range dependence, infinite variance processes, for 
example, [21]. Our interest in the subject was inspired by the appearance 
of long range dependence in occupation time fluctuations of particle sys- 
tems, specially branching systems and related superprocesses. Other types 
of long range dependence Gaussian processes connected with branching sys- 
tems with immigration are presented in [18] (without functional convergence 
proofs). 

We now give some definitions and notation. 

For the (d, a, /3)-branching particle system (with /3 < 1), let {Nt)t>o denote 
the empirical measure process, that is, Nt{A) is the number of particles in the 
set A C M.'^ at time t. Thus, A^o is a Poisson random measure with intensity 
A. The rescaled occupation time fluctuation process is defined by 



where Ft is a norming to be determined, and T is the scaling parameter 
which accelerates the time and will tend to oo. Note that ENg = A for all s, 
due to the initial Poisson condition, the criticality of the branching and the 
a-stable motion. 

Constants are written C, Ci, and so forth, with possible dependencies 
in parenthesis. (•,•) denotes pairing of spaces in duality [e.g., 5'(M^) and 
iS(M'^)]. stands for weak convergence. 

Section 2 contains the results, and Sections 3 and 4 the proofs. 



(1.4) 




t > 
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2. Results. We start by introducing the process that plays a fundamental 
role in the paper. Let M be the independently scattered (1 + /3)-stable mea- 
sure on M'^"'"^ with control measure A^+i (Lebesgue measure) and skewness 
intensity 1, that is, for each A e ^(R'^+i) such that < \d+i{A) < oo, M{A) 
is a (1 + /?)-stable random variable with characteristic function 

expj - Arf+i (^) I z| (^1 - i(sgn z) tan | (1 + /3)^ | , z eR, 

the values of M are independent on disjoint sets, and M is a-additive a.s. 
(see [26], Definition 3.3.1). Let pt(x) denote the transition density of the 
symmetric a-stable Levy process in R*^, and recall our assumption 

The process = {S.t)t>o is defined as follows. 
Definition 2.1. Let 
(2.2) it = j^^_^^{t[Q,t]{r) j\u-r{x)dv^M{drdx), t>0, 

where the integral with respect to M is understood in the sense of [26], 

(3.2) -(3.4). 

By [26] , existence of this process follows from the fact that 

(2.3) J J pu-r{x) diL^ drdx<oo. 

It can be verified that under (2.1) this integral is indeed finite (see, e.g., [17], 
Lemma A.l). 

The first main result is the following functional limit theorem for the 
process Xt defined by (1.4). 

Theorem 2.2. Assume (2.1). Let 

(2.4) Ft = t(2+/3- . 
Then 

Xt K\i 

in C{[Q,t],S' iW^)) as T ^ oo for any t > 0, where is the process defined 
by (2.2) and 
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In the next proposition we collect some basic properties of the process ^. 



Proposition 2.3. (a) ^ is (1 + (3) -stable, totally skewed to the right, 
with finite- dimensional distributions given by 



E exip{i{zi^ti H 



exp 



(2.5) 



Pu-rix) du 



.{x) du 



IT 



X tan-(l +/3) 



dr dx 



0<ti<--- <tk,zi,...,ZkeR. 

(b) ^ is self-similar with index H = {2-\- j3 — + (i), that is, 

a>0. 

(c) ^ has continuous paths (more precisely, has a continuous version). 



(^ati ) ■ • ■ ) Catfc ) — (Cii ) • • • ) Ctfc ) ) 



Property (a) follows immediately from the definition and [26] (Proposition 
3.4.2). Property (b) can be easily derived from (2.5) using the self-similarity 
of pt- Property (c) is a consequence of Theorem 2.2. 

Remark 2.4. (a) It is not hard to verify that if we put /3 = 1 in (2.5), 
we obtain the finite-dimensional distributions of the sub- fractional Brownian 
motion (multiplied by a constant) considered in [5] and [6]. This, together 
with the fact that Theorem 2.2 is an analogue of Theorem 2.2 in [6], suggests 
giving the name sub-fractional stable process to ^. See [5] concerning rela- 
tionships between sub-fractional Brownian motion and fractional Brownian 
motion, which make the name more appropriate in that case. 

(b) We think that a functional limit theorem also holds for the occupa- 
tion time fluctuation process of the (d, a, /3)-system with /3 < 1 and initial 
equilibrium state, but we have not endeavored to prove it. In this case we 
conjecture that the limit process is of the form KXrj, where ry = {'i]t)t£R is 
a self-similar, continuous, stable process with stationary increments, which 
should be a kind of fractional stable process. Moreover, by analogy with 
the covariance results for the case (3 = 1 [5], we conjecture that the process 
^ defined by (2.2) has the same distribution as the process (rjt + r]-t)t>o 
(multiplied by a constant). 
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(c) The continuity of can also be derived from the results of [22] with 
some technical work. 

The process ^ does not have independent increments and the increments 
are not stationary. In order to investigate its long range dependence, we 
introduce the following general notion. 

Definition 2.5. Let ?? be a real infinitely divisible process. For <u < 

V < s < t,T > 0, zi, Z2 let 

DTizi,Z2;u,v,s,t) 
(2.6) = |log£;e*^^^*^^''~^"^"''^^*^''^+*~''^+^''-* 

— log£^e*^^''^"~^"-' — log Ee'^^^^^^+^~^^+''^ \ . 

We define the dependence exponent k of the process r/ by 

K= inf inf sup{7 > : ^^(zi, Z2; u, t;, s, t) = o(T~''') 

zi,Z2GM. 0<u<v<s<t 

(2.7) 

as T — > 00}. 

Remark 2.6. (a) If r/ has independent increments, then k = +00. 

(b) If r] is Gaussian, then 

DTizi,Z2;U,V,S,t) = \ziZ2Cov{rjy -7]u,r]T+t-r]T+s)\- 

(c) Dt is the modulus of the codifference of the random variables zi{rjy — 
rju) and —Z2{r]T+t — VT+s), as defined in [25] (see also [26] for symmetric 
stable case). 

The second main result is the following theorem on the long range depen- 
dence of the process ^. 



Theorem 2.7. The dependence exponent of the process ^ defined by 
(2.2) is given by 

d d 

— , if either a = 2, or a <2 and (3 > , 

a d + a 

+ ifa<2andl3< ^ 

{21 



a\ a + d J d + a 



Remark 2.8. (a) Note that for /? > 1/a/2, we have P > d/{d + a), and 
for /3< (^5- l)/2, we have I3<d/{d + a). 
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(b) As we shall see in the proof, in the case (5 > d/ {d + a) the value of k 
gives the exact asymptotics of Dt, that is, 

(2.9) DT{zi,Z2]u,v,s,t) = 0{T~'^''^) asT^oo, 

provided that ziZ2> 

(c) The real valued limit process in Theorem 2.2 in [6] has the form 
C{a,d)C, where the process C (sub-fractional Brownian motion) depended 
only on d/a. In the present case all relevant parameters related to the process 
^ depend only on (3 and d/a [see (2.1), (2.4), Proposition 2.3(b) and (2.8)]. 
A natural question is whether ^ also has a form C{a,d, (3)^' , where the 
distribution of the process depends only on /? and d/a. We have not been 
able to answer this question. 

(d) The standard symmetric a-stable Levy process on M"^ is transient for 
d> a, and its degree of transience, defined as 

7 = sup{6'>0:-E;L^ <oo}, 

where L is the last exit time from an open unit ball centered at the origin, 
is given by 

d 

7 = --l 

a 

(see [13] and [27]). In the (7,/3)-plane the regions corresponding to the two 
long range dependence regimes of the process ^ are given as follows: 



fi; = 7+ l forO<7< \/2 and max/ , ^ ^ 1 < (3 < minil, —1, 

L7 + 1 7 + 2J I 7J 

K = (7 + l)fl + /3-^) 

l7'7 + 2i' 



J 1 a ■ 

for 7 > and < p < min< 

' 2 7 + 1 

and K = 7 + 1 on the separating curve /3 = , -^^-^-^ < 7 < V2. 

3. Proof of Theorem 2.2. Without loss of generality we assume r = 1. To 
start, we gather some technical facts which will be used in the proof several 
times. 

Recall that pt{-) has characteristic function e"*'^'". We denote by Tt the 
corresponding semigroup, that is, Ttf = pt * f ■ It is well known that (self- 
similarity) 

(3.1) pt{x) = t-^/"pi{xt-'/^), 

and for a <2, 

(3-2) rnw^<Pi(^)< r^u^ 
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for some positive constants ci and C2- The upper bound obviously holds also 
for a = 2. 

We will use the following slightly stronger version of (2.3), obtained by 
the same argument: 

(3.3) / ( / pu{x)du\ dx<oo, 

provided that d < a{l + 13)/ 13 [see (2.1)]. 

We will also use the following two elementary estimates: 

0<(a + 6)i+'3-ai+^-6i+^ 

(3.4) 

< (1 + /?)a^6^+'^"^ a,6>0,/3<(5< 1, 

(3.5) (a + 6)1+^^-0^+^- 6^+^ >/36^a, 6 > a > 0. 

We will employ the space-time random field approach [4], which consists 
of investigating weak convergence of the 5'(M'^+^)-random variable Xt as- 
sociated with the process Xt, defined by 

(3.6) {Xt,^)= l\xT{t)M-.t))dt, ci>e5(M'^+i). 

JO 

The main ingredients of the proof of the theorem are weak convergence of 
Xt and tightness of {Xt}t>i in ^([0 J], 5'(M'^)). 

We need the Laplace transform of Xt- Its form is given in the following 
lemma. 

Lemma 3.1. Lei $ E 5(M'^+^), ^> > 0, and denote 

1 

'^T{x,t) = — / ^{x,s)ds. 

-TT Jt/T 

Then 

^g-(^T,*> _ gjj-p J / / \ifrp(^x,T — r)vT{x,r) dr dx 
URd JO 



(3.7) 



where vt satisfies 



V 



l + P jRd Jo 



T 

v}j^^{x,r) drdx 



vt{x, t) = f Tt^r [^t(-, T - r)(l - VT{;r)) - -^4+'3(., r) 
JO I i + p 



(3.8) 



(x) dr, 
0<t<T. 
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We omit the proof of this lemma because it can be done in the same way 
as that of (3.23) in [6], using the Feynman-Kac formula and the form of the 
generating function of the branching law; see (1.1). Note that the function 
vt corresponds to v^ij,{x,T — t,t) in [6]. By the definition of vt (see (3.12) 
and (3.19) in [6]), we have 



(3.9) < < 1. 

Proposition 3.2. Let $ G 5(M"'+^), $ > 0. Then 



lim ^e-<^^'*> 

T^oo 



(3.10) 



: exp 



V 



l+P Jud. Jo 



1 r 



$(y,s) 



X / pu-r{x) duds dy 



1+/3 



drdx 



Proof. We assume that $ is of the form ^{x,t) = (p{x)ip{t), where 
(f S 5(M'^), "0 S 5(M) and (p,'ip>0. For general <I>, the proof is the same with 
slightly more complicated notation. Denote 

(3.11) x{u) = 'ip{s)ds, xt{u)=x(^^^, <fT{x) = ^(p{x). 
Obviously, 

(3.12) x{u)<C and XTiu)<C, 

for some constant C. 

By Lemma 3.1, the Laplace transform of {Xt, *&) can be written as 



(3.13) Ee"<^^'*> =exp 
where 

h{T)-- 



^ I,{T) + h{T) - -^U{T) 



1 + 13 



l+f3 



(3.14) 
(3.15) 

(3.16) 



T / rr \ 1+/3 

N %.-uVt{x)xt{T - u) du\ drdx 



h{T) 



^t{x)xt{T — r)vT{x, r) drdx, 



T 



%-u^t{x)xt{T - u) du 



1+/3 



Vrp [x,r) 



drdx. 
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In (3.15) and (3.16) vt is the solution of the equation (3.8) with ^''r(x,t) = 
fTix)xTit)- Note that by (3.8) and (3.9), /i,/2 and are nonnegative, the 
latter fact following from 



VT{x,t)< I Tt^u^T{x)xTiT -u)du. 



(3.17) 



We will show that l2{T) and /3(r) converge to as T ^ oo, and y^/i(T) 
converges to the term in the exponent in (3.10). 

We consider first Ii. Using (3.11), (2.4), and making the changes of vari- 
ables u' = ^ and r' = ^ in (3.14), we obtain 

r-l / rr \ 1+/3 

Tx(^r-u)'P{x)x{l — u) duj drdx 



Ii{T) = T^'^/°'^f^ 

^ rp{d/a)f3 



\Jo 

1 / rr 



\J0 



PT{r-u){x - y)f{y)x{'^ -u)du\ drdx. 



Using (3.1) and substituting x' = xT ^1°^ and y' = yT we arrive at 



\J0 



'Pr^uix - y) 



(3.18) 



xx{l-u)T^/^^{yTy^)dyduj 



drdx 



{fr * gT{x))^^'^ dx dr, 



where 

(3.19) fr{x) = rpr~u{x)x{l - u)du and grix) =T''l''^{xT^I''). 
Jo 

By (3.12) and (3.3), it follows that fr G Li+^(]R^) for any r e [0,1], there- 
fore, taking into account the form of gx, we have that fr * gx converges in 
L^+^(M'^) to /]gd ip{y) dy fr for any r e [0, 1] as T — > oo. Moreover, by Young's 
inequality (see, e.g., [28], Theorem 6.3.1), (3.11), (3.19) and (3.3), 



sup Wfr* grWlXp^ ll/r|li+fl||S'T||l 
re [0,1] re [0,1] 



1+/3 



<Ci 



Pu{-)du 



1+/3 



l|l+/3 ^ 



We apply the dominated convergence theorem to (3.18) to conclude that 



(3.20) 



hm h{T)= [' [ ([ ipiy)dy 



\ 1+/3 

Pr^u{x)x{^ ~ duj dxdr. 
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Recalling the definition of x [see (3.11)] and substituting v! = 1 — u,r' ■ 
1 — r, it is easy to see that (3.20) is the same as 



(3.21) 



hm /i(r)= / 



ip{y)i){s) 
X j pu-rix) duds dy 

Now we proceed to l2- 

Applying (3.17) and (3.12) to (3.15), we obtain 

hiT) <ci [ r ^t{x) rTr.uM 

JR'i Jo Jo 

Substituting u' = ^,r' = ^ and using (3.11), we obtain 



1+/9 



drdx. 



x) du drdx. 



T 

(3.22) I^{T)<Ci^[ f f\{x)TT(^,_^^^{x)dudr 

-f'-f jRd Jo Jo 



1 rr 



dx. 



Next we use (2.4), the Plancherel formula and the fact that Tsf{z) = e ^'^'"^(z) 
(^denoting Fourier transform). Then (3.22) becomes 

Ci 



h{T)< 



(3.23) 



(27r)< 



.^-l+2((d/a)/3-l)/(l+/3) 



1 r 1 



-Tr\x\^ 



-\(p{x)\^ dx dr. 



It is easy to check that under (2.1) we have -1 + 2(f /3 - 1)/(1 + /3) < 0, 
and since also a < d, it follows from (3.23) that 



(3.24) 



hm /2(r) = o. 

1 — ^oo 



It remains to prove that H{T) also converges to 0. 
We need some more notation: 



(3.25) Ji{T)=( Tr-u{^T{-)vT{-,u)){x)du 

Jw^ Jo Uo 

(3.26) J2(T)=/ / 

jR<i Jo 



drdx, 



%~u{vj. ' {■,u)){x)du 



drdx. 



By (3.8) and (3.16) 

jR'i Jo I Uo 



Tr-u^T{x)XT{T - u) du 

%^u^t{x)xt{T - u) du 
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(3.27) 

%.-u{^t{-)xt{T - u)vt{-,u)){x) du 
V 



1 + /3 

Note that by (3.9) we have 



1+/3 

%~u{v^ {■ ,u)){x) du 



drdx. 



pr 

/ %.-u^t{x)xt{T -u)du 
Jo 

(3.28) > r Tr~u{VT{-)XT{T -u)vT{;u)){x)du 



We apply (3.4) with 6 = {1 + (3)/2 and a = ai + a2, where 



ai= Tr-u{fT{-)XT{T - u)vT{-,u)){x)du, 
Jo 

02 = 7-7-5 / %--uivT~^^i-,u)){x)du, 

i + P JO 

rr 

a + b= Tr-u^T{x)xT{T - u)du, 
Jo 



[6 > by (3.28)], and then we use the estimate (oi + 02)^+^ < 2^(0}+^ + 



ag"*"^) to arrive at 
h{T) < Ci{Ji{T) + J2{T)) 

rT / rr 



+ C2 [ [ { I Tr^u{^T{-)XT{T -u)vT{-:U)){x)du 
JRd JO \J0 

^Y^Jq %-u{v].^'^{-,u)){x)dv}j 



(l+/3)/2 



%_u{'Pt{-)xt{T -u){l- vt{-,u))){x) du 



%-u{vrp {■■,u)){x)du\ drdx. 



l + f3Jo 

Using the estimate (a + 6)(^+^)/2 < a^^+^^Z^ + in the first factor under 

the integral J-^d Jq and (3.28) in the second one, then finally by the Schwarz 
inequality and (3.12), we obtain 



(3.29) /3(T) < CiiMT) + MT)) + C2^h{T){JJr{T) + JUT)) 
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It remains to prove that Ji{T) and J2{T) tend to as T ^ cx). 
By (3.17), (3.12) and (3.25), we have 



Ji{T)<Ci 



T 



< 



iR'i Jo 
7^3+2/3 

Cl 



Tr-u ^t{-) / %,-v(Pt{-) dv (x) du 



1+13 



drdx 



2+2(3 



F, 



1+/3 



dx 



2^3+2/3 
772+2/3 



X / Pv{{y — z)T ^/°')ip{z)dvdudzdy 



dx, 



where we used the definition of % and (3.1) in the last step. 

We now recah Ft [see (2.4)] and substitute x' = xT~^/°^,y' = T~^/°'y, z' - 
ji-i/q^ to obtain 

Ji(.T)<CiT-' f (f f\^{x-y)duTW-)mi+P)^^Ty-y) 
JM.'i yjR^'i Jo 

(3.30) 

X J Pv{y- z)dvT'^'''^{T^'''z)dzdyj dx. 

To simplify the notation, we introduce the functions 

"1 



f{x) = I Pu{x)du, 
Jo 

(3.31) 5i,T(a;) =rW")^/(^+^V(2^^^"^), 
52,t(2;) =T'^/"(^(r^/"x). 

It is easy to check that 

(3.32) ||gi,T||(i+/3)//3 = ||¥'ll{i+/3)//3 < 00 and ||fi(2,T||i = Hi/^lli < 00. 
In the notation of (3.31) the inequality (3.30) can be written as 

Ji(r)<CiT-i||/*(gi,T(/*92,T) 



|l+/3 
ll+/3- 



We use consecutively the Young, the Holder and again the Young inequali- 
ties, obtaining 

Ji{T)<CiT~Witf^\\9i,T{f*92,T)\\l^^ 



l+l3\\y^'T\\{l+/3)/f3 



1+/3 



1+/3||„ ul+P 
l+/3ll5'2,T|li • 
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By (3.31), (3.32) and (3.3), it follows that 
(3.33) lim Ji(T)=0. 

T — s-oo 

The term J2(T) can be dealt with in a similar manner. By (3.26), (3.17) 
and (3.12), we have 

1+13 \ 



dx 



J2{T)<Ci J^^J^ ^^{jq '^""Uo '^"-^''^^^^j {x)duj 

As in the case of Ji, we use consecutively the definition of 7^, (3.1), 
substitutions x' = xT^^/°',y' = yT~^/°',z' = zT^^/", and (2.4) to obtain 

J2(T)<Cir^-'^/°^ 

/■ r /■ /"i 

(3.34) 



Pu{x-y)du 

p„(y - z) dvT'^/''ip{T^/''z) dz j 



dx. 



Using notation (3.31) and applying the Young inequality twice, (3.34) can 
be estimated in as follows: 

J2(T) < Ciri-w°)^||/ * (/ * 52,t)'+^||S? 
<Ciri-W")'3||/||i:?||(/*52,T)^+^|ir 



By (3.32) and (3.3), we obtain 

(3.35) lim J2(T)=0. 

T — ^■oo 

By (3.33), (3.35), (3.24) and (3.29), we get 

(3.36) lim /3(r) = 0. 

T — >oo 

Putting together (3.13), (3.21), (3.24) and (3.36) finishes the proof of the 
proposition. □ 

We now pass to the tightness. We state a slightly more general result 
which includes also the lower critical dimension d= a/ (3. This is used in a 
forthcoming paper [30]. 
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Proposition 3.3. Assume that 

(3.37) if^'^-V^' 

Then the family {Xt}t>i is tight in C([0, 

Proof. The fact that the process lacks moments of order > 1 + 
/? for /3 < 1 prevents the use of standard methods for proving tightness. 
Also, the Laplace transform technique we have employed for showing weak 
convergence of the space-time random field Xt does not seem to be amenable 
to a tightness proof in the present case. Instead, we will give a proof based 
on the characteristic function of Xt- 

By Theorem 12.3 of Billingsely [1] and the theorem of Mitoma [24], it 
suffices to show that for any ip G 5(M'^) there exist constants > and 
7 > such that 

(3.38) Pi\{XT{t2),ip) - {XT{ti),^)\ >6)< 

holds for all ti,t2 G [0, 1], ti < ^2, all T > 1, and all 5>0. 

Since each cp E 5(M'^) can be written as ip = tpi — ip2, 'Pi,'P2^ S{W^),'pi, 
¥^2 ^ 0, (see, e.g., the lemma in Section 3 of [6]), it suffices to assume > 0, 
which we do from now on. So, fix (/9 > and ti,t2. 

In order to prove (3.38), we use the estimate (see, e.g., [2], Proposition 
8.29) 

P(|(Xt,^®V)I ><5) 

(3.39) 

<C5 / (l-Re(^exp{-ie(XT,y?®V')}))d0, 
Jo 

where (arguing as in the tightness proof in [7]) V' G SiM) is an approximation 
of — Stj supported on [ti,t2] such that 

x(i)=^V(s)ds 

satisfies 



(3.40) XG5(M), 0<x<MhM- 

Hence, it suffices to prove that for any x satisfying (3.40), the right-hand 
side of (3.39) (with the corresponding tp) is estimated from above by the 
right-hand side of (3.38), with constants not depending on x- To this end. 
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we define a complex-valued analogue of the function vt considered before, 
namely (using the same notation), 

(3.41) V0^T{x,t) = l-Ee-x.Y)^y-ie j^{N'^,LpT)xT{T-t + s)ds^, 6* > 0, 

where ipT, XT are given by (3.11), and is the empirical measure of the 
branching system started from a single particle at x (see (3.12) and (3.19) 
in [6]). 

Since the Feyman-Kac formula holds for complex-valued functions, the 
same procedure used before (see Lemma 3.1) shows that vq^t satisfies 



ve,T{x,t) = [ Tt^ 
(3.42) ^° 



iOipTXriT - s)(l - ve^T{-,s)) 

(x) ds 



1 + 

[cf. (3.8)], and 



(3.43) Eexp{-i9{XT,<f(^ij)} = exp{I + II}, e>0, 
where 

(3.44) I = id / ipt{x)xt{T — s)vg^T{x,s)dsdx, 

Jw^ Jo 

(3.45) // = f r vl\^{x, s) ds dx 

[cf. (3.7)]. In these equations z^~^^ = exp{(l + /?) log z} is understood in the 
sense of the principal branch of the logarithm. 
Using the inequality 

(3.46) \l-e'\<2\z\ 
if |e^| < 1, z G C, we have, by (3.43), 

(3.47) 0<l-ReEexp{-ie{XT,(p'^i')} <2{\I\ + \II\). 

Now, by virtue of the previous discussion, taking into account (3.39) and 

(3.47) , we see that (3.38) will be proved if we show that 

(3.48) \I\<C{^,^)9^{t2-ti)'+^ 
and 

(3.49) \II\ < C{ip,-f,V,P)e'+^{t2 - ti)^+^ 
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for any 7 such that 

d3 

(3.50) 0<7<l + /3-— . 

a 

To prove (3.48) and (3.49), first we observe that by (3.41) and (3.46) we 
have 

|i.»,T(i,t)l <2«-E f\N;,,pT}xTiT-t + ,)ds 

(3.51) ;° 

= 29 Tt-s(pT{x)xT{T - s)ds, 
Jo 

since E{Ng,ip) =Tsip{x) (this known fact is obtained by the usual renewal 
argument). 

Combining this and (3.44), we obtain 

(3.52) ^ 
2^2 T 











Jo Jl 


Js 



-^''''-'^\'\"x{r)drdzds, 

where, besides obvious substitutions, we have used the Plancherel formula 
as in (3.22). 

Fix 7 satisfying (3.50) and note that ^ > 1 by (3.37). 
By Holder's inequality, we have 

(3.53) < (^^'e-(^/(i-^))(--'*)l-l"dry~^(^^'x'/^(r)firy 

< (1 - 7)i^^T-^+^|zr"(^-^)(t2 - hy, 

where in the last estimate we used (3.40). 

Combining (3.52) and (3.53) and using (3.40), once again we obtain 

|/| < c{j)e^^ f |^(z)|2|zr°(i-^) dzit2 - ti)i+^ 

and this will imply (3.48) if we show that T^+^' /F^ is bounded in T > 1. To 
this end, by (2.4), we need to check that 

2(2 + /3-(d/?)/a) ^ 

' + ^ TVp 
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or 



7< 



3 + p- 2/3'^/^ 



and this inequality is indeed satisfied by (3.50) since it can be easily verified 
[using (3.37)] that 

l + /3--/3< i\o 

a 1 + p 

We now pass to the proof of (3.49). By (3.45) and (3.51), we have 
\n\<—^2^+l'9^+n / [ Ts^rM^)XT{T-r)dr] dsdx 

1+ p jRd JO Vjo / 

[by (3.1), (2.4) and (3.11)] 



C{V, 



rpd/afi 



m.'i Jo 



Jw^ 



-1/", 



JR'i \J0 



X ip(y)x{l — r) dy dr j dsdx 
Ps-r{x-y)T''l^^{T^l^y) 

xx{^ — ^)dydr\ dsdx 



<C{V,(5)e^^'^ 



Ps-r(-)x(l -r)dr 



ds||T'^/°^(ri/"-)ll}^^> 



where we have used the Young inequality in the last step. It is now clear 
that (3.49) will be proved if we show that 



(3.54) 



vo 



Ps-r{x)x{'^ -r)dr] dsdx < C{t2 - ti)^+'^. 



Denote fx{s) = Ps{x)1[q^i]{s) for any x G M"^, and g{s) = x(l — s), and put 



(3.55) 

Note that by (3.50), 

(3.56) 

and 



1 + /3 
1+/3-7' 



1 + P 
1+7' 



a 



l<P<^{l + f5), Kg, 



1 1 1 

p^q~TTp 



1. 
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By the Young inequality, we have 

"1 / /-s \ 1+/9 



^ ^ Vs^r{x)x{\ - r) drj ds < \\f, * ^ll^? < 11/ 



But ll^llg"'''^ < (^2 — ^i)'^'^'^ by (3.40) and (3.55), therefore, to finish the proof 
of (3.54) (and the proposition as weh), it suffices to show that 



where 



/ \\U\\l+^dx = A + B<oo, 
A= [ WUl'^^dx, B=[ WUl^^dx. 

J\x\>l ^ J\x\<l ^ 



l\x\>l ■J\x\<l 

By (3.1) and (3.2), we have 

A= / s-P'^/yPJs-^/'^x)ds] dx 

J\x\>l\Jo J 

<C I cix( s-prf/"+p(i+d/a) 

J\x\>l \Jo 

B is estimated analogously. We observe that from (3.56) we have 

(d/a)p — 1 d 



(l+/3)/p 

< oo. 



{{d + a)/a)p {d + a){l + P)' 

So, if we fix p such that 

(d/a)p — 1 d 
< p< 



{{d + a)/a)p ' (d + a)(l + /3)' 

and use the fact that pi{x) < C{p)pi{x) (since pi is bounded and p < 1), 
then we have, by (3.1) and (3.2), 

'\x\<l 



B<C |2.|-(l+/3){rf+a)p / g-pd/a+p{l+d/a)p < ^O. 

J|x|<i VJo / 



□ 



We still need a result on convergence of Laplace transforms, which we 
formulate as a lemma. This result is known, following from a standard ar- 
gument on analytic extensions (see, e.g., [20] for the proof in the case d=l, 
the proof for d = 2 is analogous). 

Lemma 3.4. LetQ<[3<l. 

(a) If r]n,n= 1,2,... are real random variables such that 
(3.57) lim Ee-P"^" = e^f'^^ , p > 0, 
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for some constant K > 0, then r]n converges in distribution to a random 
variable r] whose law is {1 + P) -stable, totally skewed to the right, with char- 
acteristic function 



(3.58) 



Ee'^^^ = exp<i K( cos ^(1 + /3) j f 1 - z(sgnz) tan ^(1 + f3)]\, 



zeR. 



(b) If rjn,n= 1,2,... are two-dimensional random variables and fi is a 
finite measure on S"^ = {{ui,U2) :nf +1*2 = l,ni,n2 > 0} such that 

(3.59) lim Ee-P-'^" =expl [ (/> • 2/)^+^//(dy)l, p G M^, 

n-*oo [Jg2^ J 

then r]n converges in distribution to a (1 + (3) -stable random variable r] with 
characteristic function 



\z-y 



1+/3 



^e*^-^ = exp|(^cos|(l + /?)) 
(3.60) X (^l-f(sgn(z-y))tan|(l + /3)^^(dy)|. 



z e 



(■ denotes the inner product in M^J. 

Corollary 3.5. For each $ e cS(R'^+^), 
(3.61) (Xt,^>)^C$ as CO, 

where is {1 + (3)-stable with 



Ee*«* =exp(-ifi / 
(3.62) 



1+/3 



X 1 — i s; 



^iy,s) J pu-r{x)dudsdy 
gn( [ [ $(y, s) / pu-r{x)dudsdy 

KJRd Jr Jr J 

X tan ^(1 + /3)^ dr dxj. 



where 
(3.63) 
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Proof. Proposition 3.2 and Lemma 3.4(a) imply that (3.61) holds for 
any $ > 0. This, applied to pi$i + P2^2, Pi,P2^ 0,$i,<I>2 > 0, and Lemma 
3.4(b) imply weak convergence of {{Xt,^i), (Xr,$2)) with a p obtained in 
a standard way (see, e.g., [20], proof of Theorem 5.6). Hence, (3.61) follows 
since, analogously as in the proof of Proposition 3.3, we can write any $ G 
5(M'^+^) as ^> = ^>i -^>2,4>i,^2>0. □ 

Now we can complete the proof of Theorem 2.2. By Proposition 3.3, we 
know that {Xt}t>i is tight. Let Xt„ ^ X in C([0, 1], 5'(M"')) for some 
Tn / oo. Then (Xt„, ^ (X, ^) for any $ G 5(M'^+^) (see [4]), and 

^exp{i(X, ^>)} = Eexp{i^$}. 

We now find the finite dimensional distributions of X arguing, for instance, 
as in the proof of Proposition 4.1 of [4]. Fix < ti < • • • < < 1, zi, . . . , Zfc £ 
M, (^i,...,(^fcG5(]R'^). 

Let Vmj S ^(M), — > ^tj in 5'(M) as m ^ oo, j = 1, . . . ,k. Define 

k 

^m{x,t) = ^Zjipj{x)lprn,jit); 

then 

k 

hmjX,^^)=Y,Zj{X{tj),^,), 
i=i 

and it is easily seen that the right-hand side of (3.62) (with $m instead of 
$) converges as m — > oo to 

[see (3.63)], where the distributions of are given by (2.5). This means that 
X = kI^^^~^^^ X^^, and Theorem 2.2 is proved. 

Remark 3.6. The proof of Theorem 2.2 is also valid for (3 = 1, so we 
obtain the result of [6] as a special case. Note, however, that, in contrast to 
[6], in this proof the form of the covariance of the empirical process Nt is 
not needed. 

4. Proof of Theorem 2.7. By (2.5), it is clear that it suffices to investigate 
the asymptotics of 

(4.1) D:^ = DT{l,z;u,v,s,t), z>0, 
and 

(4.2) D:^ = DTil,-z;u,v,s,t), z>0. 
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Fix 0<u<v<s<t and z > 0. The theorem wih be proved if we show 
that 

(4.3) D$ < CT~'^/" if either a = 2, or a < 2 and /3 > ^ 



d + a 



D$ < CT-W^)^ for any /3 < 5 < 1 + /3 ■ 



(4.4) 



d + a 



if a < 2, p< 



d 



d + a 

and for T sufficiently large, 

(4.5) D^yCT-'^l'', 

(4.6) > CT-^'^/")^ for any S > 1 + f3 



if a < 2, /3 < 



d + a d + a 

Here and in the sequel the constants C, Ci , and so forth are different in each 
line and may depend on d, a, (3, u, v, s, t, z, but never on T. 
Denote 

t+T 



(4.7) 



U = zl^Q^t+T]{r) / Pr'-r{x)dr' 

Jr 

rs+T 

- Z]l[0,s+T](?^) / Pr'-r{x)dr' 
J r 

( Z"*^'^ 
= ^(^l[0,s+T]('') y_^^ Pr'~rix)dr' 



t+T 



+ l(_s+T,t+T]{r) / pr'-r{x)dr' 
Jr y 

R = '^[o,v](^) / Pr'-rix)dr' -l[Q^^]{r) / pr>^r{x)dr' 



(4.8) 

= 1[0,«]('^) j Pr'-r{x)dr' + l(^^^^^{r) j Pr'-r{x)dr'. 
By (2.5), (2.6) and (4.1), we have 



(4.9) 
and, analogously, 



D-r, 



l-aan-(l + /5)) / [{U + Rf+f^ -U^+f^ - R^+f^]dxdr 

2 / jRd+l 



dxdr 



(4.10) +i('tanJ(l + /3)) / - ?7|^+^sgn(i? - [/) 

V 2 / 7Rd+i 
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Denote for brevity 








(4.11) 


f = f(x,r) = z 


rt+T 
Js+T 


Pr'^rix) dr', 


(4.12) 


gi = giix,r) = ^ 


/ Pr' 

u 


-rix) dr' , 


(4.13) 


g2 = g2{x,r) = ^ 


/•V 

I 


-rix) dr' . 


By (3.1) and (3.2), 


we have 







(4.14) fix,r) = 

(4.15) gj{x,r)< 



t-r 
s—r 

c 



\d+a ' 



(/ + r)-^/>i((r' + r)-i/"x) dr' < CT~^/'', 



(4.16) gjix,r)< pr'{x)dr', j = l,2. 
Jo 

Note that the constants C in (4.14) and (4.15) do not depend on x,r. 
By (4.7)-(4.9), 



D+ = C 



(4.17) 



r f {{f + g,r^-f'^^-g\-'')dxdr 

Jo JRd 

{{f + g2f^^-f'^^-gl-'^)dxdr. 



Using (3.4) with 5 = 1, we obtain (omitting dxdr) 



D+<C{l + (3) 



J\x\<l 



fg',+ 



J\x\>l 



fg'! 



+ 



u J\x\<l 



fg'.+ 



u J\x\>l 



fg'. 



Assume a < 2 and /5 > djid + a). Then (4.3) for D^^ follows by (4.14), 

(4.15) (for > 1) and (4.16) (for |x| < 1). If a = 2, then g^^ is integrable 
over {|x| > 1} for all /? since by the properties of the Brownian density, we 
have gj{x,r) < Cie-<^2l^l for |x| > 1 [instead of (4.15)], so we obtain (4.3) 
without any restriction on f3. 

Now assume a < 2 and P <d/{d + a). Fix P < 6 < 1 + (3 — d/{d + a), and 
apply (3.4) to (4.17). Again by (4.14)-(4.16), we have 



D+ < CiT-Wo')S 



\x\<l \J0 



Pr'{x) dr' j 



dx 
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+ (v — u) / (x) dr' I 



'|a.'|<l \Jo 
+ (v — u) [ I ,,,, w-i , „ (ix 

Hence, (4.4) for D^t follows since 1 + P - 6 < 1 and (d + a)(l + /3 - (5) > d. 

(4.3) and (4.4) for Di^ are derived in the same way. We only apply the 
following easy consequences of (3.4): 

||a - - a'+f - b^+^l < (3 + P)a'b'+^-\ 

P<S<1, a,b>0, 

I \a - sgn(a - 6) + b^+f^ - a^+^j < (1 + p)a^b^+'^^\ 

P<6<1, a,b,>0. 

We now pass to the lower estimates of D^. 
By (4.17), 

/•{u+v)/2 f 

(4.18) D+>C / {{f + g,)^+(^-f+'^-gl+f')dxdr. 
Note that for g2 defined by (4.13), if |x| < 1 and r £ [u, ^^y^], we have 

r(v~u)/2 ^ 

92{x,r)> / pj.'{x)dr > C{u,v,d,a) > 0, 

J{v-u)/A 

by (3.1). This, combined with (4.14), implies 

(4.19) 52 > / for T large, 

for x,r as above. Hence, we can apply (3.5) to (4.18) and obtain 

r-(«+l.)/2 



DJ,>C / / f{x,r)g2{x,r)dxdr 

Ju J\x\<l 

(4.20) 

r-{u+v)/2 I- 

>Ci / f{x,r)dxdr. 

Ju J\x\<l 

For x,r as above, using the equality in (4.14), we have 

f{x,r)>z / (r' + r)-'^/>i((r'+r)-^/"x)dr' 

Js-(u+v)/2 

(4.21) 
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thus obtaining (4.5). 

Finally, assume a < 2 and (5 <d/{d + a). Let ^ <£ <d/{d + a)a. 
By (4.17), 

Di>C 1^'^^""" I {U + 92r'-f'^'-9l-'')dxdr. 

(4.22) 

For x,r in the domain of integration in (4.22), we have 
92{x,r)> Z^'' "^^^(r')"'^/>i((r')"^/"x)dr' 

J{f-u)/4 



(4.23) 



>|— ) —ml 



rj.\d-\-a 



by (3.1) and (3.2). Taking into account the equality in (4.14) and (3.2), we 
see that (4.19) holds also in the present case. Inequality (3.5) applied to 
(4.22) yields 

(4.24) D±>Ci / f{x,r)g(^{x,r)dxdr. 

Ju Jl<|x|<Td/((d+<^)<»)-e 

As pi(r-i/"a;) > C2 > for T > 1 and \x\ < r'^/((rf+°)«)-^, from the equality 
in (4.14), we obtain (4.21) again. 

Combining the estimates (4.23), (4.24) and (4.21), we see that, for large 

T, 

JJ+ ^ ^rj-\[~d/ a+£{d+a))f3rp~d/ arp(d/ ({d+a)a)~£)d 
= CT~('^/")[/3+l-f^/('i+")+£a(l-/3('^+a)/rf)] 

This implies (4.6) since 1 - > and e can be made arbitrarily small. 
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